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MESSAGE LOGGING FOR RELIABLE MULTICASTING 
ACROSS A ROUTING NETWORK 

Cross -Reference to Related Applications 

This application contains subject matter which is 
5 related to the subject matter of the following United States 
patent applications, which are assigned to the same assignee 
of this application. Each of the below-listed applications 
is hereby incorporated herein by reference: 

"Routing Messages Within A Network Using The Data 
10 Content Of The Message," by Chandra et al . , filed 

November 20, 1997, serial no. 08/975,303; 

"Message Sequencing For Ordered Multicasting Of A 
Message Across A Routing Network," by Banavar et al., 
co-filed herewith, serial no. , (attorney 

15 docket no. Y0998-526) ; 

"Quiescent Reconfiguration Of A Routing Network," 
by Miller et al . , co-filed herewith, serial no. , 
(attorney docket no. Y0998-527); and 

"Non-Disruptive Reconfiguration Of A 
20 Publish/Subscribe System," by Miller et al., co-filed 

herewith, serial no. , (attorney docket no. 

Y0999-124) . 
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Technical Field 

This invention relates in general to multicasting 
messages within a network and, in particular, to logging 
messages to persistent storage within a routing network to 
5 facilitate reliable delivery thereof to one or more clients, 
Further, the invention relates to reliably multicasting a 
message within a content-based routing network irrespective 
of destination information that may be included within the 
message . 

10 Background of the Invention 

Many network environments enable messages to be 
forwarded from one site within the network to one or more 
other sites using a multicast protocol. Typical multicast 
protocols send messages from one site to one or more other 
15 sites based on information stored within a message header. 
That is, each message has two components: the message 
header, which includes the routing information, including 
destination addresses or a predefined group name that is 
associated with a fixed list of destinations, and a data 
20 content, which is the data of the message. The routing 

information is read from the message header and is used to 
send the data content of the message to the specified 
destinations . 

One example of a system that conventionally includes 
25 such a network environment is a publish/subscribe system. 

In publish/subscribe systems, publishers post messages and 
subscribers independently specify categories of events in 
which they are interested. The system takes the posted 
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messages and includes in each message header the destination 
information of those subscribers indicating interest in the 
particular message. The system then uses the destination 
information in the message to forward the message through 
5 the network to the appropriate subscribers. 

In large systems, there may be many subscribers 
interested in a particular message. Thus, a large list of 
destinations needs to be added to the message header and 
used in forwarding the message. The use of the list, which 

10 can be even longer than the message itself, can degrade 

system performance. Other approaches have included the use 
of multicast groups, in which destinations are bound to a 
group name, and then that name is included in the message 
header. The message is then sent to all those destinations 

15 bound to the name. This technique has the disadvantage of 
requiring static groups of destinations, which restricts 
flexibility in many publish/subscribe systems. 

Disclosure of the Invention 

A publish/subscribe system consists of a network of 
20 message routers (or simply routers) connected via links in 

an arbitrary graph topology. A number of clients connect to 
the periphery of this router network and either publish or 
subscribe to messages. A message consists of a number of 
attributes, which are name-value pairs. A subscription 
25 specifies a predicate on the attributes of messages. 

The router network is responsible for routing messages 
from publishers to interested subscribers based on matching 
events to subscription predicates. One embodiment of this 
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routing protocol, referred to as content-based routing, is 
described in the above-incorporated co-pending patent 
application entitled "Routing Messages Within A Network 
Using The Data Content Of The Message." Content-based 
5 routing proceeds as follows. From each router node at which 
a publisher is present, the system computes and stores a 
spanning tree to reach every other node in the network. All 
published messages from a particular publisher then follow 
the paths in the corresponding spanning tree, with each 
10 router node performing enough matching to determine which of 
its child routers should receive the published message. All 
paths preserve a first-in first-out (FIFO) ordering on 
messages from any one publisher to all subscribers. 

Briefly summarized, the present invention is directed 
15 in one aspect to providing a mechanism for multicasting 

messages in a manner which is resilient to failures in the 
network, routers, or clients (e.g., publishers/subscribers). 
Resiliency is achieved by logging messages to persistent 
storage within the network prior to delivery thereof to one 
20 or more clients. 

More particularly, provided herein is a method for 
routing messages within a network. The method includes: 
receiving a message; and routing the message to one or more 
clients of the network, the routing being based on data 
25 content of the message irrespective of any destination 
information that may be within the message, and being 
resilient to router or link failure within the network. 

In another aspect, the invention comprises a method for 
routing messages within a network which includes: receiving 
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a message; logging the message to persistent storage within 
the network; and routing the message to one or more clients 
of the network after logging the message to persistent 
storage, wherein the logging is used to ensure resiliency in 
routing the message to one or more clients of the network 
notwithstanding failure at a router or link within the 
network . 

In yet another aspect, a system of routing messages 
within a network is provided. The system includes means for 
receiving a message, and means for routing the message to 
one or more clients of the network. The routing is based on 
data content of the message irrespective of any destination 
O information that may be within the message, and the means 

ill for routing is resilient to router or link failure within 

SO 15 the network. 



10 



In still another aspect, a system of routing messages 
within a routing network is provided which includes means 
for receiving a message, means for logging the message to 
persistent storage within the routing network, and means for 

20 delivering the message to one or more clients of the network 
after logging thereof to the persistent storage. By logging 
the message to persistent storage prior to delivery thereof, 
a technique is provided which ensures resiliency to the 
routing of the message to the one or more clients 

25 notwithstanding router or link failure within the network. 

In a further aspect, an article of manufacture is 
provided which includes at least one computer usable medium 
having computer readable program code means embodied therein 
for effecting routing of messages within the network. The 
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computer readable program code means in the article of 
manufacture includes: computer readable program code means 
for causing a computer to effect receiving a message; and 
computer readable program code means for causing a computer 
5 to effect routing the message to one or more clients of the 
network, the routing being based on data content of the 
message irrespective of any destination information that may 
be within the message, and being resilient to router or link 
failure within the network, 

10 In a still further aspect, an article of manufacture is 

provided which includes at least one computer usable medium 
having computer readable program code means embodied therein 
for effecting routing of messages within a routing network. 
The computer readable program code means in the article of 

15 manufacture includes: computer readable program code means 
for causing a computer to effect receiving a message; 
computer readable program code means for causing a computer 
to effect logging the message to persistent storage within 
the routing network; and computer readable program code 

20 means for causing a computer to effect delivering the 

message to one or more clients of the network after the 
logging thereof, wherein the logging to persistent storage 
prior to delivery of the message provides resiliency to the 
routing network notwithstanding router or link failure 

25 within the network. 

To restate, provided herein is a technique for logging 
messages within a routing network itself to ensure reliable 
multicasting across the network. Although principally 
described herein in connection with data-content routing of 
30 messages, the logging technique can be applied to other 
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routing systems, e.g., subject-based routing. The technique 
is scalable through the provision of multiple loggers within 
the network. Additionally, multiple loggers can be used to 
balance the load on the system. A system implementing the 
logging technique of the present invention allows senders 
and receivers to connect anywhere in the network regardless 
of the location of the loggers. Messages are preferably 
logged only once within a router network. Again, logging of 
messages occurs within the network itself between the sender 
and the one or more clients to which the message is 
directed. 

The logging technique described herein allows a sender 
or receiver to choose the level of reliability (i.e., 
quality of service) required by either the sender or the 
clients. This choice allows a tradeoff of system resources 
versus quality of service. For a sender or receiver 
requesting "uniform delivery", the routing network either 
delivers the message to all clients or delivers the message 
to none of the clients. For a sender or receiver requesting 
"at most once delivery", the routing network detects and 
eliminates duplicate messages. "At least once delivery" 
quality of service is also provided by ensuring that loss of 
a message within the routing network is identified, and the 
sender is notified to retransmit the message. For a sender 
or receiver requesting "exactly once delivery" the routing 
network neither loses the message without notifying the 
sender nor makes a duplicate delivery of the message to a 
client. Again, the particular type of delivery can be 
selected by the sender or the receiving client. 
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Brief Description of the Drawings 



The above-described objects, advantages and features of 
the present invention, as well as others, will be more 
readily understood from the following detailed description 
5 of certain preferred embodiments of the invention, when 

considered in conjunction with the accompanying drawings in 
which : 

Fig. 1 depicts one example of a distributed network 
incorporating and using the resilient routing capability of 
10 the present invention; 

Fig. 2 depicts one example of a spanning tree used in 
accordance with the principles of the present invention; 

Fig. 3 depicts one embodiment of a distributed router 
network configured with the resilient routing capability of 
15 the present invention; 

Fig. 4 depicts one embodiment of a router for use in 
accordance with the principles of the present invention in a 
router network such as depicted in Fig. 3; 

Fig. 5 depicts one example of a router configured as a 
20 logger with message logging capability in accordance with 
the principles of the present invention; 

Fig. 6a depicts one example of a client message buffer 
used by a router having clients connected thereto in 
accordance with the principles of the present invention; 
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Fig. 6b depicts one example of a logging acknowledgment 
(LACK) message sent by a logger to a router in accordance 
with the principles of the present invention; 

Fig. 7 is a flowchart of one embodiment of a resilient 
5 message routing process in accordance with the principles of 
the present invention; 

Fig. 8 is a flowchart of one embodiment of logging 
acknowledgment (LACK) processing in accordance with the 
principles of the present invention; 

10 Fig. 9 is a flowchart of one example of processing for 

failure recovery within the routing network in accordance 
with the principles of the present invention; 

Fig. 10 is a flowchart of an at most once delivery 
quality of service in accordance with the principles of the 
15 present invention; and 

Fig. 11 is a flowchart of an at least once delivery 
quality of service in accordance with the principles of the 
present invention . 

Best Mode for Carrying Out the Invention 

20 As noted above, the present invention is in one aspect 

an extension of the routing capability disclosed in the 
above-incorporated application wherein messages are routed 
in a network based on the content of the data within the 
message. In particular, the message does not need to 

25 include destination information, such as destination 
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addresses or a group destination name. Instead, data within 
the message is used to traverse a data structure to 
determine the link or links over which the message is to be 
forwarded in order to reach the consumers (subscribers or 
5 clients) interested in the message. 

In accordance with the principles of the present 
invention, this content-based routing capability is enhanced 
for multicasting messages in a manner which is resilient to 
failures in the network, its routers, or clients. The 

10 invention is referred to herein as comprising a "reliable 

routing capability" for a routing network such as the above- 
described content-based routing network. However, those 
skilled in the art will recognize that the logging technique 
presented hereinbelow for ensuring message delivery 

15 notwithstanding a failure within the network is equally 

applicable to other routing systems, such as a subject-based 
routing system. This reliable routing capability includes 
support for four "qualities of service": (1) "uniform 
delivery" under which the routing network either delivers 

20 the message to all receivers or none of the receivers, (2) 
"at most once delivery" under which the routing network 
detects and eliminates duplicate messages, (3) "at least 
once delivery" under which the routing network detects the 
loss of a message and notifies the sender to retransmit the 

25 message, and (4) "exactly once delivery" under which the 
routing network neither loses a message without notifying 
the sender nor makes duplicate deliveries to receivers. 

One example of a distributed network 100 incorporating 
and using the reliable routing capability of the present 
30 invention is depicted in Fig. 1 and described in detail 
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below. Network 100 includes, for instance, a plurality of 
computing units 102 coupled to one another by links 104. 

Each link couples two computing units in the network, 
and each computing unit may have any number of links 
connected to it. Each link is bidirectional, i.e., a 
computing unit may send and receive messages on the link. 
Each link is also connection oriented, i.e., the computer 
units at the ends of the link are notified when the link 
fails. Furthermore, each link propagates messages in a 
first-in first-out manner. Each computing unit in the 
network is either a client computer (represented by the 
smaller ovals, such as those having addresses 101a, 101b), 
meaning that it has requested to receive messages whose 
content satisfies certain properties; or it is a router 
computer (represented by the larger ovals, such as 108a, 
108b) , meaning that it forwards messages received on one 
network link onto other links on the way to the client 
computer (s). The clients are collectively referred to 
herein as clients 101 and the routers are collectively 
referred to as routers 108. (For purposes of this 
discussion, if a single computing unit serves both as a 
router and as a client, these two separate functions will be 
considered as two computing units connected by a link.) 

Each computing unit can be any type of computing unit 
that can be included in a network. For example, it can be 
an RS/6000 computing node or any other of various types of 
computing nodes, processors, computers or systems. The 
network can also include different types of computing units 
coupled to one another via the links. The links include, 
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for instance, TCP connections over IP links, as only one 
example . 

Distributed network 100 can be included in various 
systems that require the passing of messages or data. These 
5 systems include, for instance, the internet or a content- 
based publish/subscribe system. 

Content-based publish/subscribe systems improve the 
degree of decoupling between publishers and subscribers. In 
content-based publish/subscribe systems, subscriptions are 

10 specified in terms of predicates on the posted data, rather 
than in terms of subject identifiers supplied by the 
publisher. One example of a content-based publish/subscribe 
system is described in co-pending U.S. Patent Application 
Serial No. 08/975,280, entitled "Method And System For 

15 Matching Consumers To Events," Astley et al., which is 

hereby incorporated herein by reference in its entirety. 
The examples described herein are with reference to a 
content-based subscription system. However, these are only 
examples. The present invention can be employed with other 

20 types of systems without departing from the spirit of the 
present invention . 

In one embodiment of the invention, each router 108 of 
network 100 (Fig. 1) has associated therewith a spanning 
tree, which lays out the best path (according to some 
25 criterion, such as latency) from the router to each client 
101. In this embodiment, it is assumed that routers agree 
on a common criterion for measuring distance between nodes 
in the network. There may in fact be multiple spanning 
trees. For example, alternative spanning trees may specify 
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either backup routes, or peak load routes. Herein, it is 
assumed that one spanning tree is in effect for the routing 
of any particular message. 

One example of a spanning tree, which is associated 
with router 108a (Fig. 1), is depicted in Fig. 2. As shown 
in Fig. 2, there is a path from router 102a to every other 
node in the spanning tree. A message to be routed from 
router 102a to one or more of the other nodes is routed via 
one or more of the links associated with router 102a, i.e., 
links 1-3. For example, if a message is to be routed from 
node 102a to node 101c, then link 2 is used. As a further 
example, if a message is to be forwarded toward client 101a, 
then link 3 is used. 

One embodiment for building a spanning tree from a 
network topology, that is an arbitrary graph, is described 
in detail in Introduction to Algorithms , by Cormen, 
Leiserson, Rivert, Chapter 24, pp 498-513, Published by MIT 
Press (1990), which is hereby incorporated herein by 
reference in its entirety. 

In addition to the spanning tree associated with each 
router, each router has a routing table. The routing table 
includes an entry for each client computer in the network. 
Each entry of the routing table associates a client address 
with the identifier of the network link constituting the 
next segment on the path in the spanning tree from the 
router to the client. For a router with d network links, 
each such link identifier is an integer between 1 and d. 
For instance, the client having address 101a has a 
corresponding link identifier of 3 (see Fig. 2) . 
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Each routing table is constructed via information from 
the network topology (e.g., the client addresses) and hence 
from the corresponding spanning tree (e.g., the link 
identifiers), in a known manner. 

5 By way of example, Fig. 3 depicts one embodiment of a 

publish/subscribe system, generally denoted 300, to employ 
resilient message routing in accordance with the principles 
of the present invention. System 300 includes a network of 
routers 302 connected via links 303 in an arbitrary graph 

10 topology. A number of clients connected to the periphery of 
this router network either publish messages or subscribe to 
messages. Those clients publishing messages comprise 
publishers 304, while those clients subscribing to messages 
comprise subscribers 306. The router network is responsible 

15 for routing messages from a publisher 304 to interested 

subscribers 306 based on matching messages to subscription 
predicates . 

This protocol, known as content-based routing, is 
described in detail in the initially-incorporated co-pending 

20 patent application entitled "Routing Messages Within A 

Network Using The Data Content Of The Message. " Briefly 
explained, from each router node at which a publisher is 
present, the system computes and stores a spanning tree to 
reach every other node in the network. All published 

25 messages from the publisher follow the paths in that 

spanning tree, with each router node performing enough 
matching to determine which of its child routers should 
receive the message. 
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The present invention achieves reliability of routed 
messages by saving messages to persistent storage within the 
router network and retrieving and redelivering the message 
whenever there is a failure in the network. Publishers and 
5 subscribers that need reliability of messages may specify a 
quality of service parameter, e.g., "uniform delivery". 
Uniform delivery is provided for ensuring delivery of a 
message to all active subscribers notwithstanding failure in 
the network, e.g., the routers, or the links. Special 

10 routers in the router network 300 are designated logging 

nodes or loggers 310 and support the ability to log messages 
to stable storage. When there is at least one subscriber 
needing logging, the routing algorithm ensures that messages 
are routed to a logger. When reliable delivery is required 

15 by at least one publisher or subscriber, the routing 

protocol incorporates message logging in accordance with the 
principles of the present invention. 

Briefly explained, messages originally sent by a 
publisher are assigned unique ids by the first router node 

20 receiving the message. All routers (including logging 

nodes) receiving a published message thereafter note the 
message's unique message id and the source node (i.e., 
publisher) . Routing information comprising the nodes to 
which the message will be routed is then computed based on 

25 message content and stored, e.g., indexed by using the 

message id. Routing computation always includes the nearest 
logging node. Messages are then forwarded to applicable 
neighboring routers and delivered to any subscribing clients 
to that node not requiring reliable delivery. Messages to 

30 the node's subscribing clients requiring reliable delivery 
are not delivered, but instead buffered in a client message 
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buffer (see Fig. 6a) for later delivery as explained further 
below. 

Fig. 4 depicts one embodiment of a router node for use 
in accordance with the principles of the present invention. 
5 When a message arrives at router 400 it is stored into a 
message table 402 using the unique message identifier. 
Information stored includes the source node identification 
as well as the neighboring nodes to which the message is to 
be forwarded. These neighboring nodes are calculated by a 

10 content routing computation component 404 after the message 
has been received. Computation component 404 takes the 
message and based upon stored subscriptions returns a set of 
destinations or links upon which the message should be 
forwarded. Again, in one embodiment , this computation is 

15 content dependent and can be accomplished as described in 
the above-incorporated co-pending application. However, 
this component can be implemented in other ways, e.g., 
subject-based routing. In accordance with the principles of 
the present invention, router 400 also includes a reliable 

20 routing component 406 and logging acknowledge (LACK) 
received table 408 and LACK send table 409, which are 
described further below. 

As noted, pursuant to this invention one or more 
selected routers in the network also serve as logging nodes. 

25 Fig. 5 depicts one embodiment of a logging node or logger 
which comprises a router 450 and persistent or stable 
storage 460, such as a file system or a database. When a 
published message is received at any logger node, the logger 
node performs the following logging functions in addition to 

30 the standard routing functions described above. 
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1. 



The message is assigned a next logging number in 



sequence . 



2. 



The message is written to disk 460 using its 
logging number. 

A logging acknowledgment (LACK) is sent to those 
routers to which the message was routed, as well 
as back towards the source node of the message. A 
LACK message (Pig. 6b) includes the message id, 
the logger id, and the logging number. 



5 



3. 



10 



When a LACK message is received by a router, it looks 



up the routing information for the original message, sends 
the LACK message down the links the original message was 
sent excluding the link from which the LACK message was 
received. In addition, the router forwards the LACK towards 

15 the source of the original message, and deletes the routing 
information stored in the message table for that message. 
If the message table was lost due to the router failing and 
recovering, the LACK message is forwarded along all links on 
the spanning tree with the logger at the root, excluding the 

20 link from which the LACK message was received. Also, if the 
original message corresponding to the LACK was buffered 
waiting to be delivered to one or more subscribing clients 
requiring uniform delivery, the message will now be 
delivered. 

25 In support of automatic recovery from failure, each 

router node also stores (pursuant to the present invention) 
the following information while processing LACK messages. 
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• For each logger node from which a LACK has been 
received, the latest logging number that has been 
received for that logger. 

• For each link on which a LACK has been forwarded, 
5 the latest logging number that has been sent. 

This information is stored into the LACK tables 408, 
409 (Fig. 4) for the router. Since pursuant to this 
invention FIFO links are assumed between a logger and any 
given router, receiving (or sending) a LACK for a logging 
10 number N implies that all relevant LACKs for logging numbers 
less than N have also been received (or sent) . Note that 
non-first-in first-out (non-FIFO) links are made FIFO for 
the present invention, for example, using standard 
techniques such as the TCP protocol. 

15 As implied in the above description, three processing 

phases are implemented in one embodiment in accordance with 
the principles of the present invention. In a first phase, 
messages are routed from a publisher to all nodes in the 
network, but not necessarily delivered to all subscribers, 

20 depending upon whether uniform delivery has been specified 
by the publisher or the subscribers. In a second phase, 
messages are delivered from the routing nodes based upon an 
acknowledgment received from the logger node (i.e., the LACK 
message) . This phase two processing and use of the loggers 

25 ensures uniform delivery of messages requiring reliable 

routing. A third phase is entered whenever a node or link 
fails within the network. This phase involves a request for 
and a replaying of messages that were lost due to the 
failure . 



Y0998-525 



-18- 



Fig. 7 depicts one embodiment of phase one message 
routing processing in accordance with the present invention. 
This processing is implemented at each node of the network. 
At a given node, a published message is received 700 and 
5 recorded into the node's message table 402 (Fig. 4) using 

the message id and source node information provided with the 
message 710. The node's content routing computation 404 
(Fig. 4) next determines neighboring node routing 
information which is also stored into the message table 720. 
10 Again, this computation is based on data content of the 

message when employing a routing system such as described in 
the initially-incorporated patent application. 

The message is next forwarded to applicable neighboring 
nodes 730 as determined by the content routing computation, 

15 and delivered to subscribing clients not requiring uniform 
delivery 740. The subscribing clients comprise clients 
coupled to the particular router node processing the 
published message. For example, certain subscriber clients 
to a particular router node may require uniform delivery, 

20 while others may not. Thus, the process allows those 
clients not requiring uniform delivery to immediately 
receive the published message. Again, as used herein, 
"uniform delivery" comprises a system guarantee that 
subscribers who have requested uniform delivery receive the 

25 message only if all other subscribers requesting uniform 

delivery also receive the message. Otherwise, the message 
is not forwarded to any subscriber requesting the uniform 
delivery. Either every subscriber receives the published 
message or no subscriber of uniform delivery receives the 

30 published message. 
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In order to deliver messages to clients requesting 
uniform delivery, the router node proceeds to buffer the 
message for later delivery to those clients 750. If the 
router node is a logger 760, the message is assigned a 
5 logging number and written to disk 770. A logging 
acknowledgment (LACK) is then sent to the applicable 
neighboring router nodes (computed in step 720) , as well as 
back to the source node providing the message now held in 
the message table 780. This completes 7 90 this embodiment 
10 of the message routing phase. 

Fig. 8 depicts an approach for processing LACK messages 
in accordance with the principles of the present invention. 
Each router node in the network implements this processing 
upon receipt of a LACK message 800. First, the routing 

15 information for the original message is determined from the 
node's message table 810. The LACK message is then sent to 
those nodes where the original message was sent minus the 
neighboring node from which the LACK message was received 
and plus the original message's source node 820. The 

20 routing information corresponding to this message id is 
deleted from the node's message table 830, and, if the 
message was buffered in the client message buffer table 
(such as depicted in Fig. 6a) for delivery to one or more 
clients, then the message is now delivered. 

25 Essentially, the process comprises determining whether 

there is a message id in the uniform delivery table 
corresponding to the id of the LACK message being processed. 
If so, then the message is retrieved from the uniform 
delivery table and delivered to the one or more clients of 

30 the node requesting uniform delivery. Thus, uniform 
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delivery means that the message is buffered in the client 
message buffer of the node and then delivered once the 
logging acknowledgment signal is received back from a 
logging node of the network. 

5 In a third phase, a routing system in accordance with 

the principles of the present invention enters a failure 
recovery process whenever a router detects failure of, for 
example, its parent node, or the link to its parent node. 
The detecting child router notifies a configuration manager, 

10 which we assume tracks the configuration of the router 
network. The configuration manager replies with a 
replacement parent node. This new parent has the same 
parent and children as the failed node. Implementation of 
this monitoring for node failure and replacement of a parent 

15 node are within the capabilities of one of ordinary skill in 
the art. 

Note that a configuration manager is a "network system 
service" that is responsible for maintaining information 
about the structure, function, and status of a network 

20 system. This includes information such as the network 

topology, nodes with certain properties (such as loggers) , 
quality of service offered nodes in the network, etc. A 
system' s configuration may be statically fixed or 
dynamically changing. A configuration manager may be 

25 centralized or distributed. Network system management 

products such as IBM's Tivoli TME-10 contain configuration 
management components. From the perspective of the current 
invention, the system uses the topology information in the 
configuration manager when nodes or links fail and 

30 substitutes nodes to re-establish these failed components. 
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Referring to Fig. 9, in one embodiment, the failure 
recovery phase of the present invention begins with 
detection of failure of the parent node, or a link to the 
parent node 900. The configuration manager or system 
5 replaces the failed node 910, and child nodes of the failed 
node notify the new parent of the latest logging numbers 
received from each logger of the system 920. The new parent 
collects these responses into a sequence and forwards a 
request for retransmission (RTREQ) for this range of 
10 messages to its parent node 930. Upon receiving an RTREQ 
message, each router narrows the set of requested messages 
based on its logging sequence vector for the child link from 
which the RTREQ was received, after which the RTREQ message 
is forwarded on to the next node 940. 



15 Upon receiving an RTREQ message at a logger node, the 

logger node narrows the set of requested messages and 
forwards the message onto the rest of the tree, while also 
finding all messages in the narrowed set of messages that it 
has logged and retransmits them to the new parent 950. When 

20 the new parent receives re-transmitted messages it requested 
from a logging router, it matches the messages that fall in 
the persistence range and delivers them to the appropriate 
children nodes 960. After all such messages have been 
delivered, the new parent node begins to forward new 

25 messages to its children, thereby preserving the FIFO 
property of the protocol. 

Optionally, each router node may cache a set of 
recently sent messages. When used, the cache may reduce the 
number of messages requested in an RTREQ message or may 
30 eliminate the need for an RTREQ message. 
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To summarize, using the above protocol a message is 
guaranteed to be received by all subscribers requiring the 
"uniform delivery" quality of service, or none at all will 
receive the message. However, if a publisher crashes and 
5 recovers and as a result republishes a message, the message 
may be delivered multiple times to subscribers. The present 
invention avoids this problem using the following 
augmentation to the uniform delivery protocol. The 
augmentation is referred to as "at most once delivery" of 
10 messages, and one embodiment of the process is depicted in 
Fig. 10. 

Clients desiring duplicate detection include a client- 
specific unique identifier with each message before 
forwarding the message up towards the logger node as in the 

15 original protocol 1000. Upon receipt of a data message, the 
logging engine checks to see if the node has previously seen 
the client-specific unique identifier 1010. If it has, the 
node logs the fact that it will abort this message, and 
transmits a logging abort (LABORT) message through the tree 

20 instead of a LACK message 1020. Each LABORT message still 
has a logging sequence number so that, in case of failures, 
routing nodes will receive possibly lost LABORT messages 
and, therefore, eventually remove routing information saved 
for the duplicate message. Upon receipt of a LABORT message 

25 at a node, the routing information for the original message 
is determined from the node's message table 1030. The 
LABORT message is then sent to those nodes where the 
original message was sent minus the neighboring node from 
which the LABORT message was received plus the original 

30 message's source node 1040. The routing information 

corresponding to this message id is deleted from the node's 
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message table 1050, and, if the message was buffered in the 
client message buffer table for delivery to one or more 
clients, then the message is deleted 1060. 



Using the above protocol, a message is guaranteed to be 
5 received by all subscribers requiring the "uniform delivery" 
quality of service if the message reaches a logger node. It 
may also happen that a message is lost before it reaches a 
logger, due to failure of a router or link in route to a 
logger. In this case, the message is not at all delivered 

10 to uniform delivery subscribers. It is also desirable for 
publishers to become aware of the fact that a message has 
not reached a logger node, and if so, republish the message, 
thereby guaranteeing the delivery of the message "at least 
once" to every subscriber. The present invention handles 

15 this as depicted in Fig. 11. 

Basically, a publisher can detect whether a message has 
been delivered to "uniform delivery" subscribers by 
subscribing to its own messages under the "uniform delivery" 
quality of service 1100. If the publisher receives the 

20 message, then it has been delivered to the uniform 

subscribers 1110. However, if the publisher does not 
receive the message within a certain time out period, it can 
query a logger node whether the published message has been 
logged 1120. If the reply to this query is received by the 

25 publisher before the original published message, it can be 
inferred, due to the FIFO property of links, that the 
original published message has been lost before it reached 
the logger node 1130. In this case, the message can be 
republished 1140. 
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As a further variation, "exactly once" delivery of 
messages can be ensured by using in combination the 
protocols for at most once delivery and at least once 
delivery of messages. A message is delivered exactly once 
5 to each subscriber in the face of: (1) publisher failures 
and recoveries; (2) node and link failures before a message 
gets to a logger; and (3) node and link failures after a 
message gets to a logger and before delivery to subscribers. 
These are respectively accomplished as follows: (1) a 

10 publisher includes a unique id in a message with the help of 
which the system detects duplicate publications of a message 
and removes the duplicate, as described above; (2) the 
publisher makes sure that a message reaches a logger by 
subscribing to its own messages as well as by querying the 

15 logger for lost messages, as described above; and (3) 

detecting node and link failures, re-establishing these 
nodes and links, and replaying lost messages from one or 
more loggers, as described above. 

To summarize, those skilled in the art will note from 
20 the above description that in a publish/subscribe system 
supporting content-based subscription, a method to route 
messages based on content has been provided in a manner that 
is resilient to router or link failure. The 

publish/subscribe system may be realized as a router network 
25 connected in an arbitrary graph topology, except that the 
links from a logger node to any router are assumed to be 
FIFO. Further, a method to deliver a message to either all 
subscribers requesting a "uniform delivery" quality of 
service, or to none of them, is described above. Variations 
30 on this "uniform delivery" quality of service outlined above 
include an "at most once" quality of service, an "at least 
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once" quality of service, and an "exactly once" quality of 
service . 

The present invention can be included, for example, in 
an article of manufacture (e.g., one or more computer 
5 program products) having, for instance, computer usable 
media. This media has embodied therein, for instance, 
computer readable program code means for providing and 
facilitating the capabilities of the present invention. The 
articles of manufacture can be included as part of the 
10 computer system or sold separately. 

Additionally, at least one program storage device 
readable by machine, tangibly embodying at least one program 
of instructions executable by the machine, to perform the 
capabilities of the present invention, can be provided. 

15 The flow diagrams depicted herein are provided by way 

of example. There may be variations to these diagrams or 
the steps (or operations) described herein without departing 
from the spirit of the invention. For instance, in certain 
cases, the steps may be performed in differing order, or 

20 steps may be added, deleted or modified. All of these 

variations are considered to comprise part of the present 
invention as recited in the appended claims. 

While the invention has been described in detail herein 
in accordance with certain preferred embodiments thereof, 
25 many modifications and changes therein may be effected by 
those skilled in the art. Accordingly, it is intended by 
the appended claims to cover all such modifications and 



Y0998-525 



changes as fall within the true spirit and scope of the 
invention. 




Claims 



1 



1. A method for routing messages within a network, 
said method comprising: 



2 



3 



receiving a message; and 



4 



routing said message to one or more clients of 
said network, said routing being based on data content 
of said message irrespective of any destination 
information that may be within said message, and being 
resilient to router or link failure within said 



5 



6 



7 



8 



9 



network . 



1 



2. 



The method of claim 1, wherein said network 



2 comprises a publish/subscribe system supporting content- 

3 based subscription, said one or more clients comprise 

4 subscribers, and wherein said routing comprises delivering 

5 said message to all subscribers requesting a uniform 

6 delivery quality of service or if unable to deliver said 

7 message to all of said subscribers requesting uniform 

8 delivery, delivering said message to none of said 

9 subscribers requesting uniform delivery. 

1 3. The method of claim 2, wherein said delivering 

2 said message to all subscribers requesting uniform delivery 

3 comprises delivering said message to all subscribers 

4 requesting uniform delivery notwithstanding failure at one 

5 or more routers or links of said network, said delivering 

6 comprising storing said message to persistent storage at a 

7 logging node of said network prior to providing said message 

8 to said subscribers requesting uniform delivery. 
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1 4. The method of claim 1, wherein said routing 

2 comprises logging said message at at least one logging node 

3 within said network before delivering said message to said 

4 one or more clients of said network, said logging comprising 

5 storing said message in persistent storage, 

1 5. The method of claim 4, further comprising 

2 subsequent to said logging of said message, sending a 

3 logging acknowledgment to at least one router of said 

4 network routing said message, and upon receipt of said 

5 logging acknowledgment at said at least one router, 

6 delivering said message to a client thereof, said client 

7 requiring uniform delivery and comprising one client of said 

8 one or more clients. 

1 6. The method of claim 5, further comprising 

2 buffering said message at said at least one router of said 

3 network routing said message, said buffering occurring prior 

4 to said storing of said message at said persistent storage 

5 and when passing said message through said at least one 

6 router to said at least one logging node. 

1 7. The method of claim 5, wherein said network 

2 comprises a plurality of routers coupled together, one of 

3 said routers comprising said logging node having said 

4 persistent storage associated therewith, said logging 

5 comprising employing said logging node having said 

6 persistent storage associated therewith to store said 

7 message and to thereafter send said logging acknowledgment 

8 back to each router of said network responsible for routing 

9 said message. 
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1 8. The method of claim 1, wherein said network 

2 comprises a spanning tree and wherein said method further 

3 comprises providing a logging node within said spanning tree 

4 for logging said message to persistent storage during 

5 routing of said message to said one or more clients of said 

6 network . 

1 9. The method of claim 8, wherein said routing 

2 comprises employing said logging of said message to 

3 persistent storage to ensure a uniform delivery quality of 

4 service of said message to said one or more clients of said 

5 network notwithstanding failure of one or more routers or 

6 links within said network. 

1 10. The method of claim 1, wherein said network 

2 comprises a spanning tree having a plurality of routers, 

3 said method further comprising detecting failure of a router 

4 within said tree before completing routing of said message 

5 to said one or more clients of said network, reconfiguring 

6 said tree to replace said failed router with a new router, 

7 and automatically generating a request for retransmission of 

8 said message. 

1 11. The method of claim 10, further comprising prior 

2 to said detecting of said failure, logging said message 

3 within persistent storage of said network and issuing a 

4 logging acknowledgment confirming storage of said message to 

5 at least one router of said tree through which said message 

6 is routed to said one or more clients. 
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1 12. The method of claim 10, wherein said automatically 

2 generating said request for retransmission of said message 

3 occurs if said new router detects from one or more of its 

4 child routers a logging number associated with said message, 

5 said logging number having been received in said logging 

6 acknowledgment confirming storage of said message. 

1 13. The method of claim 1, wherein said routing 

2 further comprises determining within said network whether 

3 said message comprises a duplicate message to said one or 

4 more clients of said network, and if so, aborting said 

5 duplicate message such that said message is delivered to 

6 said one or more clients at most once. 

0 1 14. The method of claim 1, further comprising 

;f 5 2 automatically informing a sender of said message when the 

frj 3 message has been lost within the network to allow the sender 

4 to retransmit said message for routing to said one or more 

m 5 clients of said network so that said message is delivered at 

H 6 least once to said one or more clients. 

w 

v« 
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1 15. The method of claim 1, wherein said routing 

2 comprises logging said message at at least one logging node 

3 within said network before delivering said message to said 

4 one or more clients of said network, said logging comprising 

5 storing said message into persistent storage, and wherein 

6 said method further comprises subsequent to said logging of 

7 said message, sending a logging acknowledgment to at least 

8 one router of said network routing said message, and upon 

9 receipt of said logging acknowledgment at said at least one 

10 router of said network routing said message, looking up 

11 routing information for said message from a message table 

12 maintained at said at least one router, then sending said 

13 logging acknowledgment across said network using said looked 

14 up routing information, and thereafter deleting said routing 

15 information from said message table. 
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1 16 . A method for routing messages within a routing 

2 network, said method comprising: 

3 receiving a message; 

4 logging the message to persistent storage within 

5 the routing network; and 

6 after said logging, delivering said message to one 

7 or more clients of said network, wherein said logging 

8 to persistent storage prior to delivery of said message 

9 to said one or more clients of said network provides 

10 resiliency to said routing notwithstanding router or 

11 link failure within said network. 

1 17. The method of claim 16, wherein said logging 

2 comprises storing said message in said persistent storage at 

3 a logging node within said routing network before said 

4 delivery of said message to said one or more clients of said 

5 network . 

1 18. The method of claim 17, further comprising sending 

2 a logging acknowledgment to at least one router of said 

3 network routing said message after said logging of said 

4 message to said persistent storage, and upon receipt of said 

5 logging acknowledgment at said at least one router of said 

6 network routing said message, delivering said message to a 

7 client thereof, said client thereof requiring uniform 

8 delivery and comprising one client of said one or more 

9 clients. 
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1 19. The method of claim 16, wherein said network 

2 comprises a spanning tree and wherein said method further 

3 comprises providing a logging node within said spanning tree 

4 for said logging of said message to persistent storage 

5 during routing of said message to said one or more clients 

6 of said network. 

1 20. The method of claim 19, wherein said routing 

2 comprises employing said logging of said message to 

3 persistent storage to ensure a uniform delivery quality of 

4 service of said message to said one or more clients of said 

5 network notwithstanding failure of one or more routers or 

6 links within said network. 

0 1 21. The method of claim 16, wherein said routing 

..Hi 

2 network comprises a spanning tree having a plurality of 

f;Q 3 routers, said method further comprising detecting failure of 

4 a router within said tree before completing routing of said 

fit 5 message to said one or more clients of said network, 

H 6 thereafter reconfiguring said tree to replace said failed 

p* ; 7 router with a new router, and automatically generating a 

kj 8 request for retransmission of said message from said 

::J? 9 persistent storage. 

1 22. The method of claim 16, further comprising 

2 determining within said routing network whether said message 

3 comprises a duplicate message to said one or more clients of 

4 said network, and if so, aborting said duplicate message 

5 such that said message is delivered to said one or more 

6 clients at most once. 
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1 23. The method of claim 16, further comprising 

2 automatically informing a sender of said message when said 

3 message has been lost within the routing network to allow 

4 the sender to retransmit the message for routing to said one 

5 or more clients of said network so that said message is 

6 delivered at least once to said one or more clients. 
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1 24. A system of routing messages within a network, 

2 said system comprising: 

3 means for receiving a message; and 

4 means for routing said message to one or more 

5 clients of said network, said routing being based on 

6 data content of said message irrespective of any 

7 destination information that may be within said 

8 message, and wherein said means for routing is 

9 resilient to router or link failure within said 
10 network. 

1 25. The system of claim 24, wherein said network 

2 comprises a publish/subscribe system supporting content- 

3 based subscription, and wherein said one or more clients 

4 comprise subscribers, with said message being received from 

5 a publisher. 

1 26. The system of claim 25, wherein said means for 

2 routing comprises means for delivering said message to all 

3 subscribers requesting a uniform delivery quality of service 

4 or if unable to deliver said message to all of said 

5 subscribers requesting uniform delivery, for delivering said 

6 message to none of said subscribers requesting uniform 

7 delivery. 

1 27. The system of claim 26, wherein said means for 

2 routing delivers said message to said subscribers requesting 

3 uniform deliver notwithstanding failure at one or more 

4 routers or links of said network, and wherein said system 

5 further comprises means for logging said message to 

6 persistent storage prior to delivery thereof to said 

7 subscribers requesting uniform delivery. 
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1 28. The system of claim 24, wherein said means for 

2 routing comprises means for logging said message to at least 

3 one logging node within said network before delivering said 

4 message to said one or more clients of said network, said 

5 means for logging comprising means for storing said message 

6 in persistent storage. 

1 29. The system of claim 28, further comprising means 

2 for sending a logging acknowledgment to at least one router 

3 of said network routing said message after said means for 

4 logging stores said message in persistent storage, and 

5 wherein said system further comprises, at said at least one 

6 router of said network routing said message, means for 

7 delivering said message to a client thereof upon receipt of 

8 said logging acknowledgment, said client requiring uniform 

9 delivery and comprising one client of said one or more 
10 clients. 

1 30. The system of claim 29, further comprising means 

2 for buffering said message at said at least one router of 

3 said network routing said message, said buffering occurring 

4 prior to said storing of said message at said persistent 

5 storage by said means for logging. 

1 31. The system of claim 29, wherein said network 

2 comprises a plurality of routers coupled together, one of 

3 said routers comprising said logging node having said 

4 persistent storage associated therewith, and wherein said 

5 means for logging comprises means for employing said logging 

6 node having said persistent storage associated therewith to 

7 store said message and to thereafter send said logging 

8 acknowledgment back to each router of said network 

9 responsible for routing said message. 
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1 32. The system of claim 24, wherein said network 

2 comprises a spanning tree and wherein said system further 

3 comprises a logger node within said spanning tree for 

4 logging said message to persistent storage during routing of 

5 said message to said one or more clients of said network. 

1 33. The system of claim 24, wherein said means for 

2 routing comprises means for employing said logger node to 

3 log said message to persistent storage to ensure a uniform 

4 delivery quality of service of said message to said one or 

5 more clients of said network notwithstanding failure of one 

6 or more routers or links within said network. 

1 34. The system of claim 24, wherein said network 

2 comprises a spanning tree having a plurality of routers, and 

3 further comprising means for detecting failure of a router 

4 within said tree before completing routing of said message 

5 to said one or more clients of said network, and means for 

6 reconfiguring said tree to replace said failed router with a 

7 new router, and means for automatically generating a request 

8 for retransmission of said message. 

1 35. The system of claim 34, further comprising means 

2 for logging said message within persistent storage of said 

3 network and for issuing a logging acknowledgment confirming 

4 storage of said message to at least one router of said tree 

5 through which said message is routed to said one or more 

6 clients. 
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1 36. The system of claim 35, wherein said means for 

2 automatically generating a request for retransmission of 

3 said message comprises means for detecting a logging number 

4 associated with said message stored at one or more child 

5 routers of said new router. 

1 37. The system of claim 24, wherein said means for 

2 routing further comprises means for determining within said 

3 network whether said message comprises a duplicate message 

4 to said one or more clients of said network, and if so, for 

5 aborting said duplicate message such that said message is 

6 delivered to said one or more clients at most once. 

1 38. The system of claim 24, further comprising means 

Q 2 for automatically informing a sender of said message when 

P*t 3 said message has been lost within said network to allow the 

4 sender to retransmit said message for routing to said one or 

5 more clients of said network so that said message is 
fij 6 delivered at least once to said one or more clients. 
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1 39. A system of routing messages within a routing 

2 network, said system comprising: 

3 means for receiving a message; 

4 means for logging the message to persistent 

5 storage within the routing network; and 

6 means for delivering said message to one or more 

7 clients of said network after said logging of said 

8 message to persistent storage, wherein said logging to 

9 persistent storage prior to delivery of said message to 

10 said one or more clients of said network provides 

11 resiliency to said routing notwithstanding router or 

12 link failure within said network. 

1 40. The system of claim 39, wherein said means for 

2 logging comprises means for storing said message in said 

3 persistent storage at a logging node within said routing 

4 network before said delivery of said message to said one or 

5 more clients of said network. 

1 41. The system of claim 40, further comprising means 

2 for sending a logging acknowledgment to at least one router 

3 of said network routing said message after said logging of 

4 said message to said persistent storage, and means for 

5 thereafter delivering said message to a client of said at 

6 least one router of said network routing said message, said 

7 client requiring uniform delivery and comprising one client 

8 of said one or more clients. 
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1 42. The system of claim 39, wherein said routing 

2 network comprises a spanning tree, and said means for 

3 logging comprises a logging node within said spanning tree 

4 for logging said message to persistent storage during 

5 routing of said message to said one or more clients of said 

6 network . 

1 43. The system of claim 42, wherein said means for 

2 routing comprises means for employing said logging of said 

3 message to persistent storage to ensure a uniform delivery 

4 quality of service of said message to said one or more 

5 clients of said network notwithstanding failure of one or 

6 more routers or links within said network. 

0 1 44. The system of claim 39, wherein said routing 

Snl 2 network comprises a spanning tree having a plurality of 

CO 3 routers, and wherein said system further comprises means for 

f Z 4 detecting failure of a router within said tree before 

MBS 

Ffj 5 completing routing of said message to said one or more 

^ 6 clients of said network, and for thereafter configuring said 

n 7 tree to replace said failed router with a new router, and 

W 8 for automatically generating a request for retransmission of 

% 9 said message from said persistent storage. 

1 45. The system of claim 39, further comprising means 

2 for determining within said routing network whether said 

3 message comprises a duplicate message to said one or more 

4 clients of said network, and if so, for aborting said 

5 duplicate message such that said message is delivered to 

6 said one or more clients at most once. 
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1 46. The system of claim 39, further comprising means 

2 for automatically informing a sender of said message when 

3 said message has been lost within said routing network to 

4 allow the sender to retransmit the message for routing to 

5 said one or more clients of said network so that said 

6 message is delivered at least once to said one or more 

7 clients. 



m 

% 

f"1 



C3 



Y0998-525 



1 47. A system for routing messages comprising: 

2 a network adapted to receive and log a message to 

3 persistent storage; and 

4 said network comprising one or more routers 

5 adapted to route said message to one or more clients of 

6 said network, wherein said routing of said message by 

7 said one or more routers is based on data content of 

8 said message irrespective of any destination 

9 information that may be within the message, and is 

10 resilient to router or link failure within the network. 
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48. A system for routing messages comprising: 



2 a network adapted to receive a message; 

3 a logger node within said network for logging said 

4 message to persistent storage; and 

5 said network comprising one or more routers for 

6 delivering said message to one or more clients of said 

7 network, wherein said logging of said message to 

8 persistent storage occurs prior to delivery of said 

9 message to said one or more clients of said network, 

10 thereby providing resiliency to said routing 

11 notwithstanding router or link failure within said 
O 12 network. 

ru 
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49. An article of manufacture, comprising: 

at least one computer usable medium having 
computer readable program code means embodied therein 
for effecting routing of messages within a network, the 
computer readable program code means in the article of 
manufacture comprising: 

computer readable program code means for 
causing a computer to effect receiving a message; 
and 

computer readable program code means for 
causing a computer to effect routing said message 
to one or more clients of said network, said 
routing being based on data content of said 
message and being resilient to router or link 
failure within said network. 
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50. An article of manufacture, comprising: 



2 at least one computer usable medium having 

3 computer readable program code means embodied therein 

4 for effecting routing of messages within a routing 

5 network, the computer readable program code means in 

6 the article of manufacture comprising: 

7 computer readable program code means for 

8 causing a computer to effect receiving a message; 

9 computer readable program code means for 

10 causing a computer to effect logging said message 

11 to persistent storage within the routing network; 

12 and 

13 computer readable program code means for 

14 causing a computer to effect delivering said 

15 message to one or more clients of said network 

16 after said logging thereof, wherein said logging 

17 to persistent storage prior to delivery of said 

18 message to one or more clients of said network 

19 provides resiliency to said routing 

20 notwithstanding router or link failure within said 

21 network. 
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MESSAGE LOGGING FOR RELIABLE MULTICASTING 
ACROSS A ROUTING NETWORK 



Abstract of the Disclosure 

A technique for reliably multicasting a message within 
5 a router network is provided. At least one special router 
in the network has associated persistent storage for logging 
a message being routed to one or more clients. When a 
message is received at this logger node, the logger places 
the message in persistent storage and sends a logging 

10 acknowledgment back to those routers to which the message 

was originally routed, as well as back towards the source of 
the message. The logger acknowledgment includes the message 
id, the logger id, and a logging number. When the logger 
acknowledgment is received by a router, it looks up the 

15 routing information from the original message, and sends the 
acknowledgment to those neighboring routers to which the 
original message was sent, excluding the link from which the 
logger acknowledgment was received. If the original message 
corresponding to the logger acknowledgment was buffered at 

20 this router waiting to be delivered, the message is then 
delivered to its client nodes. Processes for recovering 
from node or link failure within the router network, and for 
at most once delivery of messages, at least once delivery of 
messages, and exactly once delivery of messages are also 

25 provided. 
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